Motion Segment Decomposition of RGB-D Sequences for Human Behavior Understanding

International audienceIn this paper, we propose a framework for analyzing and understanding human behavior from depth videos. The proposed solution first employs shape analysis of the human pose across time to decompose the full motion into short temporal segments representing elementary motions. Then, each segment is characterized by human motion and depth appearance around hand joints to describe the change in pose of the body and the interaction with objects. Finally , the sequence of temporal segments is modeled through a Dynamic Naive Bayes classifier, which captures the dynamics of elementary motions characterizing human behavior. Experiments on four challenging datasets evaluate the potential of the proposed approach in different contexts, including gesture or activity recognition and online activity detection. Competitive results in comparison with state of the art methods are reported

3-D Human Action Recognition by Shape Analysis of Motion Trajectories on Riemannian Manifold

International audienceRecognizing human actions in 3D video sequences is an important open problem that is currently at the heart of many research domains including surveillance, natural interfaces and rehabilitation. However, the design and development of models for action recognition that are both accurate and efficient is a challenging task due to the variability of the human pose, clothing and appearance. In this paper, we propose a new framework to extract a compact representation of a human action captured through a depth sensor, and enable accurate action recognition. The proposed solution develops on fitting a human skeleton model to acquired data so as to represent the 3D coordinates of the joints and their change over time as a trajectory in a suitable action space. Thanks to such a 3D joint-based framework, the proposed solution is capable to capture both the shape and the dynamics of the human body simultaneously. The action recognition problem is then formulated as the problem of computing the similarity between the shape of trajectories in a Riemannian manifold. Classification using kNN is finally performed on this manifold taking advantage of Riemannian geometry in the open curve shape space. Experiments are carried out on four representative benchmarks to demonstrate the potential of the proposed solution in terms of accuracy/latency for a low-latency action recognition. Comparative results with state-of-the-art methods are reported

ShapeDBA: Generating Effective Time Series Prototypes using ShapeDTW Barycenter Averaging

Time series data can be found in almost every domain, ranging from the medical field to manufacturing and wireless communication. Generating realistic and useful exemplars and prototypes is a fundamental data analysis task. In this paper, we investigate a novel approach to generating realistic and useful exemplars and prototypes for time series data. Our approach uses a new form of time series average, the ShapeDTW Barycentric Average. We therefore turn our attention to accurately generating time series prototypes with a novel approach. The existing time series prototyping approaches rely on the Dynamic Time Warping (DTW) similarity measure such as DTW Barycentering Average (DBA) and SoftDBA. These last approaches suffer from a common problem of generating out-of-distribution artifacts in their prototypes. This is mostly caused by the DTW variant used and its incapability of detecting neighborhood similarities, instead it detects absolute similarities. Our proposed method, ShapeDBA, uses the ShapeDTW variant of DTW, that overcomes this issue. We chose time series clustering, a popular form of time series analysis to evaluate the outcome of ShapeDBA compared to the other prototyping approaches. Coupled with the k-means clustering algorithm, and evaluated on a total of 123 datasets from the UCR archive, our proposed averaging approach is able to achieve new state-of-the-art results in terms of Adjusted Rand Index.Comment: Published in AALTD workshop at ECML/PKDD 202

Space-time Pose Representation for 3D Human Action Recognition

International audience3D human action recognition is an important current challenge at the heart of many research areas lying to the modeling of the spatio-temporal information. In this paper, we propose representing human actions using spatio-temporal motion trajectories. In the proposed approach, each trajectory consists of one motion channel corresponding to the evolution of the 3D position of all joint coordinates within frames of action sequence. Action recognition is achieved through a shape trajectory representation that is learnt by a K-NN classifier, which takes benefit from Riemannian geometry in an open curve shape space. Experiments on the MSR Action 3D and UTKinect human action datasets show that, in comparison to state-of-the-art methods, the proposed approach obtains promising results that show the potential of our approach

Compréhension de comportements humains 3D par l'analyse de forme de la posture et du mouvement

L'émergence de capteurs de profondeur capturant la structure 3D de la scène et du corps humain offre de nouvelles possibilités pour l'étude du mouvement et la compréhension des comportements humains. Cependant, la conception et le développement de modules de reconnaissance de comportements à la fois précis et efficaces est une tâche difficile en raison de la variabilité de la posture humaine, la complexité du mouvement et les interactions avec l'environnement. Dans cette thèse, nous nous concentrons d'abord sur le problème de la reconnaissance d'actions en représentant la trajectoire du corps humain au cours du temps, capturant ainsi simultanément la forme du corps et la dynamique du mouvement. Le problème de la reconnaissance d'actions est alors formulé comme le calcul de similitude entre la forme des trajectoires dans un cadre Riemannien. Les expériences menées sur quatre bases de données démontrent le potentiel de la solution en termes de précision/temps de latence de la reconnaissance d'actions. Deuxièmement, nous étendons l'étude aux comportements plus complexes en analysant l'évolution de la forme de la posture pour décomposer la séquence en unités de mouvement. Chaque unité de mouvement est alors caractérisée par la trajectoire de mouvement et l'apparence autour des mains, de manière à décrire le mouvement humain et l'interaction avec les objets. Enfin, la séquence de segments temporels est modélisée par un classifieur Bayésien naïf dynamique. Les expériences menées sur quatre bases de données évaluent le potentiel de l'approche dans différents contextes de reconnaissance et détection en ligne de comportements.The emergence of RGB-D sensors providing the 3D structure of both the scene and the human body offers new opportunities for studying human motion and understanding human behaviors. However, the design and development of models for behavior recognition that are both accurate and efficient is a challenging task due to the variability of the human pose, the complexity of human motion and possible interactions with the environment. In this thesis, we first focus on the action recognition problem by representing human action as the trajectory of 3D coordinates of human body joints over the time, thus capturing simultaneously the body shape and the dynamics of the motion. The action recognition problem is then formulated as the problem of computing the similarity between shape of trajectories in a Riemannian framework. Experiments carried out on four representative benchmarks demonstrate the potential of the proposed solution in terms of accuracy/latency for a low-latency action recognition. Second, we extend the study to more complex behaviors by analyzing the evolution of the human pose shape to decompose the motion stream into short motion units. Each motion unit is then characterized by the motion trajectory and depth appearance around hand joints, so as to describe the human motion and interaction with objects. Finally, the sequence of temporal segments is modeled through a Dynamic Naive Bayesian Classifier. Experiments on four representative datasets evaluate the potential of the proposed approach in different contexts, including recognition and online detection of behaviors

Comprensione del comportamento umano 3D attraverso l’analisi di forma del movimento e della posa

The emergence of RGB-D sensors providing the 3D structure of both the scene and the human body offers new opportunities for studying human motion and understanding human behaviors. However, the design and development of models for behavior recognition that are both accurate and efficient is a challenging task due to the variability of the human pose, the complexity of human motion and possible interactions with the environment. In this thesis, we address this issue in two main phases by differentiating behaviors according to their complexity. We first focus on the action recognition problem by representing human action as the trajectory of 3D coordinates of human body joints over the time, thus capturing simultaneously the body shape and the dynamics of the motion. The action recognition problem is then formulated as the problem of computing the similarity between shape of trajectories in a Riemannian framework. Experiments carried out on four representative benchmarks demonstrate the potential of the proposed solution in terms of accuracy/latency for a low-latency action recognition. Second, we extend the study to activities by analyzing the evolution of the human pose shape to decompose the motion stream into short motion units. Each motion unit is then characterized by the motion trajectory and depth appearance around hand joints, so as to describe the human motion and interaction with objects. Finally, the sequence of temporal segments is modeled through a Dynamic Naive Bayesian Classifier. Experiments on four representative datasets evaluate the potential of the proposed approach in different contexts, including gesture or activity recognition and online activity detection.L’émergence de capteurs de profondeur capturant la structure 3D de la scène et du corps humain offre de nouvelles possibilités pour l’étude du mouvement et la compréhension des comportements humains. Cependant, la conception et le développement de modules de reconnaissance de comportements à la fois précis et efficaces est une tâche difficile en raison de la variabilité de la posture humaine, la complexité du mouvement et les interactions avec l’environnement. Dans cette thèse, nous abordons cette question en deux étapes principales en différenciant les comportements en fonction de leur complexité. Nous nous concentrons d’abord sur le problème de la reconnaissance d’actions en représentant la trajectoire du corps humain au cours du temps, capturant ainsi simultanément la forme du corps et la dynamique du mouvement. Le problème de la reconnaissance d’actions est alors formulé comme le calcul de similitude entre la forme des trajectoires dans un cadre Riemannien. Les expériences menées sur quatre bases de données démontrent le potentiel de la solution en termes de précision/temps de latence de la reconnaissance d’actions. Deuxièmement, nous étendons l’étude aux activités en analysant l’évolutionde la forme de la posture pour décomposer la séquence en unités de mouvement. Chaque unité de mouvement est alors caractérisée par latrajectoire de mouvement et l’apparence autour des mains, de manière à décrire le mouvement humain et l’interaction avec les objets. Enfin, laséquence de segments temporels est modélisée par un classifieur Bayésien naif dynamique. Les expériences menées sur quatre bases de données évaluent le potentiel de l’approche dans différents contextes comme la reconnaissance de gestes ou d’activités et la détection en ligne d’activités.La diffusione di sensori RGB-D capaci di fornire la struttura 3D sia della scena che del corpo umano offre nuove opportunità per studiare i movimenti dell’uomo e capire i suoi comportamenti. Tuttavia, la progettazione e lo sviluppo di modelli per il riconoscimento dei comportamenti che siano tanto accurati quanto efficienti è un problema competitivo a causa della variabilità delle pose, della complessità del moto e delle possibili interazioni con l’ambiente. In questa tesi si affronta il problema in due passi principali, differenziando i comportamenti in base alla loro complessità. Si pone l’attenzione inizialmente sul problema di riconoscere azioni rappresentandole come traiettorie di coordinate 3D dei giunti del corpo nel tempo, catturando al tempo stesso la forma e le dinamiche di moto. Il problema del riconoscimento delle azioni è poi riformulato come il problema di calcolare le similarità tra la forma delle traiettorie in un manifold Riemanniano. Gli esperimenti effettuati su quattro benchmark dimostrano il potenziale della soluzione proposta in termini di accuratezza/latenza del riconoscimento di azioni. Lo studio è poi esteso al riconoscimento di attività analizzando l’evoluzione della forma delle pose per decomporre il flusso di moto in unità di moto. Ogni unità di moto è quindi caratterizzata dalla traiettoria di moto e da una descrizione della profondità nell’intorno dei giunti delle mani, in modo da descrivere il moto e le interazioni con oggetti. Infine, la sequenza di segmenti temporali è modellata attraverso un classificatore Dynamic Naive Bayesian. Il potenziale dell’approccio proposto è valutato su esperimenti con quattro dataset in contesti diversi, inclusi il riconoscimento di gesti e attività e rilevamento di azioni online

Multi-level Motion Analysis for Physical Exercises Assessment in Kinaesthetic Rehabilitation

International audienceAnalyzing and understanding human motion is a major research problem widely investigated in the last decades in various application domains. In this work, we address the problem of human motion analysis in the context of kinaesthetic rehabilitation using a robot coach system which should be able to learn how to perform a rehabilitation exercise as well as assess patients' movements. For that purpose, human motion analysis is crucial. We develop a human motion analysis method for learning a probabilistic representation of ideal movements from expert demonstrations. A Gaussian Mixture Model is employed from position and orientation features captured using a Microsoft Kinect v2. For assessing patients' movements, we propose a real-time multi-level analysis to both temporally and spatially identify and explain body part errors. This allows the robot to provide coaching advice to make the patient improve his movements. The evaluation on three rehabilitation exercises shows the potential of the proposed approach for learning and assessing kinaesthetic movements

Novel Generative Model for Facial Expressions Based on Statistical Shape Analysis of Landmarks Trajectories

International audienceWe propose a novel geometric framework for analyzing spontaneous facial expressions, with the specific goal of comparing, matching, and averaging the shapes of landmarks trajectories. Here we represent facial expressions by the motion of the landmarks across the time. The trajectories are represented by curves. We use elastic shape analysis of these curves to develop a Riemannian framework for analyzing shapes of these trajectories. In terms of empirical evaluation, our results on two databases: UvA-NEMO and Cohn-Kanade CK+ are very promising. From a theoretical perspective, this framework allows formal statistical inferences, such as generation of facial expressions

Recognition of Activities of Daily Living via Hierarchical Long-Short Term Memory Networks

International audienceIn order to offer optimal and personalized assistance services to frail people, smart homes or assistive robots must be able to understand the context and activities of users. With this outlook, we propose a vision-based approach for understanding activities of daily living (ADL) through skeleton data captured using an RGB-D camera. Upon decomposition of a skeleton sequence into short temporal segments, activities are classified via a hierarchical two-layer Long-Short Term Memory Network (LSTM) allowing to analyse the sequence at different levels of temporal granularity. The proposed approach is evaluated on a very challenging daily activity dataset wherein we attain superior performance. Our main contribution is a multi-scale, temporal dependency model of activities, founded on a comparison of context features that characterize previous recognition results and a hierarchical representation with a low-level behaviour-unit recognition layer and a high-level units chaining layer